We use a set of cosmological N-body simulations to investigate the structural shape of galaxy-sized cold dark matter (CDM) halos. Unlike most previous work on the subject - which dealt with shapes as measured by the inertia tensor - we focus here on the shape of the gravitational potential, a quantity more directly relevant to comparison with observational probes. A further advantage is that the potential is less sensitive to the effects of substructure and, as a consequence, the isopotential surfaces are typically smooth and well approximated by concentric ellipsoids. Our main result is that the asphericity of the potential increases rapidly towards the center of the halo. The radial trend is more pronounced than expected from constant flattening in the mass distribution, and reflects a strong tendency for dark matter halos to become increasingly aspherical inwards. Near the center the halo potential is approximately prolate ((c/a)_0=0.72 +/- 0.04, (b/a)_0=0.78 +/- 0.08), but it becomes increasingly spherical in the outer regions. The principal axes of the isopotential surfaces remain well aligned, and in most halos the angular momentum tends to be parallel to the minor axis and perpendicular to the major axis. This suggests that galactic disks may form in a plane where the potential is elliptical and where its ellipticity varies rapidly with radius. This can result in significant deviations from circular motion in systems such as low surface brightness galaxies (LSBs), even for relatively minor deviations from circular symmetry. Simulated long-slit rotation curves can appear similar to those of LSBs often cited as evidence for constant density "cores". This suggests that taking into account the 3D shape of the dark mass distribution might help to reconcile such evidence with the cuspy mass profile of CDM halos.